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Background / Context: 

According to the recent administrations of the National Assessment of Edueation 
Progress (NAEP) reading test, low-ineome children seore approximately three-fourths of a 
standard deviation lower, on average, than middle- and upper-ineome ehildren in fourth- and 
eighth-grade (National Center for Edueation Statistics, 2007). Although there are many 
underlying eauses of ineome-based disparities in reading aehievement, soeial seientists have 
shown that low-ineome ehildren tend to fall behind their eeonomieally disadvantaged peers 
during the summer months (Cooper, Nye, Charlton, Eindsay, & Greathouse, 1996; Entwisle, 
Alexander, & Olson, 2000). Eor example, in a meta-analysis that examined the impaet of 
summer vacation on fall test scores. Cooper and his colleagues (1996) found that low-ineome 
ehildren underwent larger summer losses in reading eomprehension {d= -21) than middle- 
ineome ehildren (£/= -.14). 

Why do low-income ehildren appear to lose more ground in reading achievement during 
the summer months? To address this question, soeial seientists have focused on inequalities in 
ehildren’ s opportunities to read and to learn at home during the summer. Numerous empirieal 
studies indieate low-income and working-elass families own fewer books and spend less time 
discussing books with their parents than ehildren from middle-income families (Burkam, Ready, 
Eee, & LoGerfo, 2004; Entwisle, Alexander, & Olson, 2000; Lareau, 1989). Given limited 
opportunities to read and diseuss books at home, many low-ineome ehildren are at-risk of falling 
behind in reading during summer vaeation. To address this problem, several reeent summer 
reading interventions have foeused on providing ehildren with more books and seaffolding 
reading experienees at home during the summer months. Although some of these home-based 
summer reading intervention studies have generated positive findings, no study has pooled 
results from these interventions to date. Thus, little is known about the overall effeetiveness of 
sehool- and home -based summer reading interventions. 

Purpose / Objective / Research Question / Focus of Study: 

Sinee the publieation of Cooper et al’s (2000) eomprehensive meta-analysis of summer 
sehool, there has been an inereasing interest in understanding the impaet of summer reading 
interventions on student outcomes. There are two major reasons why an updated review of 
summer reading interventions is needed. Eirst, Cooper et al. (2000) underseored the major 
design flaws in previous evaluations of summer sehools. In their review, they eoneluded that the 
“ambiguity assoeiated with a laek of random assignment is the single greatest threat to the 
eonelusions we have drawn from the summer sehool database” (p. 103). In our review, 
therefore, we ineluded experimental and quasi-experimental studies with high internal validity 
and exeluded single group pre-post evaluations, whieh were ineluded in Cooper et al.’s (2000) 
meta-analysis of summer sehool. 

Seeond, the enaetment of the No Child Eeft Behind Aet of 2001 eneouraged 
polieymakers to implement supplemental edueational serviees after sehool and during the sehool 
to help elose aehievement disparities. Many of these serviees take plaee in ehildren’ s home 
settings. In addition, sinee the publication of the 2000 National Reading Panel report, 
researehers have designed and implemented seaffolded summer reading interventions that are 
designed to inerease ehildren’s aeeess to books and promote effeetive independent use of 
eomprehension strategies. Given these poliey innovations, we hypothesized that the researeh 
literature would inelude more reeent and rigorous evaluations of summer literaey interventions 
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that are both school- and home -based and that focus explicitly on improving reading 
achievement. 

The purpose of this meta-analytic review is twofold. First, we estimate the impact of K-8 
summer literacy interventions on reading achievement. We measure reading achievement using 
a composite outcome and specific measures of word reading and fluency, vocabulary, and 
comprehension. Second, we examine the moderating role of the socioeconomic characteristics 
of the sample on a composite reading outcome and reading comprehension. Although there is a 
growing literature on the negative impact of summer vacation on low-income children’s reading 
achievement, little is known about the moderating role of children’s socioeconomic status on the 
effects of interventions. This review updates a comprehensive meta-analysis of summer school 
(Cooper et ah, 2000) and also extends prior work by including interventions that take place in the 
home setting. 

Key construct definitions. Summer literacy interventions are designed to improve 
children’s reading achievement through academic tasks that are implemented in home or school 
settings during summer vacation (McCombs et al., 2011). According to the RAND Research 
Reading Group’s heuristic for reading comprehension, comprehension is shaped by the 
characteristics of readers, texts, activities, and the sociocultural context. Summer literacy 
interventions are typically focused on remediating the skills of K-8 students or preventing 
reading loss among low-income children, using texts tied to school curricula or using popular 
tradebooks that are selected by children or matched to children’s interests. 

In addition, summer reading interventions can take place in either school or home 
settings. In a comprehensive meta-analysis of summer school programs. Cooper et al. (2000) 
noted that most summer school programs are designed to remediate academic weaknesses in the 
core subjects of reading or mathematics. More recently, researchers and policymakers have also 
designed and implemented more cost-effective home-based summer literacy interventions 
(Allington et al., 2010). Home-based literacy interventions often encourage children to engage 
in independent book reading activities during the summer months. Some home-based summer 
literacy interventions also scaffold home book-reading with teacher or parent - scaffolded 
activities (White & Kim, 2008). 

Research Questions: 

We are still in the process of coding all the studies that will be included in our meta- 
analytic review. Our preliminary analysis is based on 27 experimental and quasi-experimental 
studies. For the 2010 conference, we will finish coding all the studies and update our results. 

Our goal in this proposal is to address three related research questions: (1) What is the 
impact of summer literacy interventions on a composite (aggregate) measure of reading 
achievement and measures of word reading (decoding and oral reading fluency), vocabulary, and 
comprehension? (2) Does the type of summer intervention — school-based and home-based — 
moderate treatment effects? (3) Does the socioeconomic status of the sample moderate the 
effects of school-based interventions on reading comprehension? 

Research Design: 

Meta-anaiysis. The goal of meta-analysis is to combine the results of independent studies 
and to identify potential study-level moderators that explain variability in treatment effects 
(Cooper, 2010). To conduct a meta-analysis, each study-level treatment effect must be converted 
to a standardized mean difference, or, effect size. In this study, we computed Cohen’s d for each 
study (i.e., the difference between the treatment and control group divided by the pooled 
standard deviation). We used a shifting unit of analysis to ensure that effect sizes were 
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independent. Thus, for the reading composite measure, we created a mean reading outcome for 
each study by aggregating effect sizes within each study. For the analyses involving specific 
reading measures, we used the one effect size per study that was motivated by the research 
question. 

Data Collection and Analysis: 

Inclusion Criteria. Because our primary goal was to assess the causal impact of summer 
literacy interventions on student outcome measures, we included only experimental or quasi- 
experimental studies. In addition, we sought to determine whether certain program elements 
tend to have greater impact on students’ reading achievement. As such, we were open to 
studying a diverse set of summer programs offered in a variety of settings. We include studies of 
organized programs that 1) took place over the summer vacation, and 2) served K-8 students. 
These criteria encompass (a) traditional Summer reading interventions in settings outside 
children’s homes, including public school districts, programs run by nonprofits in community 
centers, or by public libraries, and (b) home-based Summer reading interventions that provide 
learning resources to students for use in the home. 

We did not search for studies on learning programs taking place during the intersession of 
a year-round, or “balanced,” school year calendar for two reasons: 1) the nine-month school 
calendar is more common than the balanced school year calendar, and 2) while much research 
documents that students’ achievement scores decline over summer vacation, the research base on 
the academic effects of a balanced school calendar is less conclusive (Cooper, Valentine, 
Charlton, & Melson, 2003). We excluded early childhood and high school programs because 
these programs tend to have different goals compared to K-8 programs. 

Literature Review, in June 2011, we began our literature search using three strategies: 1) 
searching electronic databases and targeted internet sites, 2) reviewing relevant reference lists, 
and 3) soliciting research reports from targeted states and school districts. We restricted all 
searches to studies published after August 1998 because this was the final month included in 
Cooper et al.’s literature search for their 2000 meta-analysis. 

Electronic Database and internet Searches. We searched the electronic databases of 
Academic Search Premier, Education Abstracts, ERIC, PsycINFO, EconLit, and ProQuest 
Dissertations and Theses. Our searches contained two sets of key words or phrases; the first set 
was designed to identify studies that met our programmatic inclusion criteria (“summer 
program*,” “summer school*,” “summer reading,” “summer literacy,” “summer enrichment,” 
“summer remedia*,” “summer instruction*,” “summer education*,” “summer learning”) and the 
second set was designed to narrow the results to studies more likely to meet our methodological 
inclusion criteria ("*experiment*," "control*," “regression discontinuity,” "compared," 
"comparison," “field trial*,” "effect size,” "evaluation”). These searches yielded 1691 results, 
which we exported to Refworks for review and elimination of duplicates. We also searched the 
public online databases of Child Trends LINKS, What Works Clearinghouse, and the Harvard 
Eamily Research Project’s Out-of-School Time Database. We then read each study’s abstract 
and downloaded the complete study when appropriate. To date, 27 studies have been coded and 
included in this preliminary analysis. 

Coding Studies, in our preliminary analysis, we developed a protocol to code each study 
for design (randomized experiments or quasi-experimental designs), sample characteristics (more 
than 50% low-income), intervention characteristics (school- or home-based summer reading 
interventions), and effect size characteristics (word reading = decoding and oral fluency, 
vocabulary, reading comprehension, and composite reading). 
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Findings/ Results: 

Statistical analyses. We made an a priori deeision to employ a random effeets model in 
analyzing the data. Because summer reading interventions vary along a number of dimensions 
and because we were interested in making inferences back to the population of studies from 
which our studies were sampled, we used a random effects model to pool effect sizes. The 
random effects model includes both a within-study weight (inverse of the study variance) and a 
between study-variance component. We also conducted moderator analyses to examine whether 
a study-level characteristic was associated with variability in effect sizes. 

Combined effects for total reading composite, word reading, vocabulary, and 
comprehension. Our preliminary analysis consisted of 27 experimental or quasi-experimental 
evaluations of summer reading interventions. The grand study-level mean on the reading 
composite measure was .045 (Cl = .028, .132), which was statistically significant. The Q- 
statistic of 44.14 (/?< .01) was also significant and the I-squared value of 41% indicated low to 
moderate heterogeneity among effect sizes. Inspection of specific outcomes indicated a 
statistically significant and positive mean effect size for word reading, d= .178 (Cl = .061, .295) 
and the 6 ?-statistic allowed us to reject the null hypothesis of homogenous effects O 23 = 39.98, p 
< .05). Similarly, the effect size for reading comprehension was also positive, d= .27 (Cl = .087, 
.453). The (/-statistic of 55.19 {k = 14) and I-squared statistic of 75% indicated moderate to 
high heterogeneity in effect sizes. On the basis of these findings and a priori hypotheses about 
the diversity of summer literacy interventions, we conducted moderator analyses to identify 
study-level characteristics that explained the variability in reading comprehension effect sizes. 

Moderator variables for reading comprehension. First, the socioeconomic status of the 
children in the summer reading interventions was a significant moderator of effects on reading 
comprehension. Samples with more than 50% low-income children yielded a larger mean effect 
size, d= .409 (Cl = .173, .646), than samples with a mix of income levels, d= -.062, (Cl = -.256, 
.133), a(k=l) = 5.17,/7 < .05. Second, we found no differences in mean effect sizes for home- 
based interventions, d= .255 (Cl = -.03, .54), and school-based interventions, d= .289 (CI=.035, 
.543), Q\,{k= 1) = .03, n.s. 

We also conducted separate analyses for school-based and home -based summer reading 
interventions. There were too few studies to examine whether socioeconomic status moderate 
effects in home -based programs. However, for school-based programs, studies with more than 
50% low-income children yielded a larger mean effect size, d= .53 (Cl = .161, .905), than 
studies with samples that had a mix of income levels, d= -.062 (Cl = -.256, .133), Qh{k= 1) = 
5.23,/7<.05. 

Conclusions: 

Although the findings from this meta-analysis are preliminary, they provide suggestive 
evidence that summer reading interventions can improve K -8 students’ reading achievement, 
both in word reading and comprehension. The results also indicate that studies with a majority 
of low-income children yielded greater benefits on measures of reading comprehension than 
studies with socioeconomically mixed samples. In particular, school-based summer reading 
interventions generated significantly larger gains for studies with mostly low-income children. 

In our final analysis, we hope to extend this preliminary analysis in two ways. First, we will test 
the robustness of this finding by expanding the pool of coded studies and, and second, we will 
probe whether study design explains variation in treatment effects. 
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